Hinge loss

 ( Loss Functions ) 

In machine learning, the hinge loss is a loss function used for training classifiers. The hinge loss is used for "maximum-margin" classification, most notably for support vector machines (SVMs).

For an intended output and a classifier score , the hinge loss of the prediction is defined as

$\backslash ell(y)\; =\; \backslash max(0,\; 1-t\; \backslash cdot\; y)$

Note that $y$ should be the "raw" output of the classifier's decision function, not the predicted class label. For instance, in linear SVMs, $y\; =\; \backslash mathbf\{w\}\; \backslash cdot\; \backslash mathbf\{x\}\; +\; b$, where $(\backslash mathbf\{w\},b)$ are the parameters of the hyperplane and $\backslash mathbf\{x\}$ is the input variable(s).

When and have the same sign (meaning predicts the right class) and $|y|\; \backslash ge\; 1$, the hinge loss $\backslash ell(y)\; =\; 0$. When they have opposite signs, $\backslash ell(y)$ increases linearly with , and similarly if , even if it has the same sign (correct prediction, but not by enough margin).

The Hinge loss is not a proper scoring rule.

---

Extensions While binary SVMs are commonly extended to multiclass classification in a one-vs.-all or one-vs.-one fashion,

(2025). 9783540263067 ISBN 9783540263067

it is also possible to extend the hinge loss itself for such an end. Several different variations of multiclass hinge loss have been proposed. For example, Crammer and Singer defined it for a linear classifier as

$\backslash ell(y)\; =\; \backslash max(0,\; 1\; +\; \backslash max\_\{y\; \backslash ne\; t\}\; \backslash mathbf\{w\}\_y\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{w\}\_t\; \backslash mathbf\{x\})$,

where $t$ is the target label, $\backslash mathbf\{w\}\_t$ and $\backslash mathbf\{w\}\_y$ are the model parameters.

Weston and Watkins provided a similar definition, but with a sum rather than a max:

$\backslash ell(y)\; =\; \backslash sum\_\{y\; \backslash ne\; t\}\; \backslash max(0,\; 1\; +\; \backslash mathbf\{w\}\_y\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{w\}\_t\; \backslash mathbf\{x\})$.

In structured prediction, the hinge loss can be further extended to structured output spaces. Structured SVMs with margin rescaling use the following variant, where denotes the SVM's parameters, the SVM's predictions, the joint feature function, and the Hamming loss:

$\backslash begin\{align\}$

\ell(\mathbf{y}) & = \max(0, \Delta(\mathbf{y}, \mathbf{t}) + \langle \mathbf{w}, \phi(\mathbf{x}, \mathbf{y}) \rangle - \langle \mathbf{w}, \phi(\mathbf{x}, \mathbf{t}) \rangle) \\

                & = \max(0, \max_{y \in \mathcal{Y}} \left( \Delta(\mathbf{y}, \mathbf{t}) + \langle \mathbf{w}, \phi(\mathbf{x}, \mathbf{y}) \rangle \right) - \langle \mathbf{w}, \phi(\mathbf{x}, \mathbf{t}) \rangle)
     

\end{align}.

---

Optimization The hinge loss is a convex function, so many of the usual convex optimizers used in machine learning can work with it. It is not differentiable, but has a subgradient with respect to model parameters of a linear SVM with score function $y\; =\; \backslash mathbf\{w\}\; \backslash cdot\; \backslash mathbf\{x\}$ that is given by

$\backslash frac\{\backslash partial\backslash ell\}\{\backslash partial\; w\_i\}\; =\; \backslash begin\{cases\}$

-t \cdot x_i & \text{if } t \cdot y < 1, \\
0            & \text{otherwise}.
     

\end{cases}

However, since the derivative of the hinge loss at $ty\; =\; 1$ is undefined, Smoothness versions may be preferred for optimization, such as Rennie and Srebro's

$\backslash ell(y)\; =\; \backslash begin\{cases\}$

\frac{1}{2} - ty & \text{if} ~~ ty \le 0, \\ \frac{1}{2} (1 - ty)^2 & \text{if} ~~ 0 < ty < 1, \\ 0 & \text{if} ~~ 1 \le ty \end{cases}

or the quadratically smoothed

$\backslash ell\_\backslash gamma(y)\; =\; \backslash begin\{cases\}$

\frac{1}{2\gamma} \max(0, 1 - ty)^2 & \text{if} ~~ ty \ge 1 - \gamma, \\ 1 - \frac{\gamma}{2} - ty & \text{otherwise} \end{cases}

suggested by Zhang. The modified Huber loss $L$ is a special case of this loss function with $\backslash gamma\; =\; 2$, specifically $L(t,y)\; =\; 4\; \backslash ell\_2(y)$.

---

See also

Post Comment